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Register a multitude of modality components, 
where each modality component handles an 
interface modality for an application 
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Register an application module 
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Initiate an application from a thin client; 
Application submits its application conditions 
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Thin client submits its device configuration, 
including the modality interfaces that it supports 
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Activate the modality components for the 
modalities supported by the thin client and 
connect the modality components to the device 
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Run the inference engine against the list of 
activation conditions and the current state of 
the shared memory area 
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One of the application's activation conditions is 
selected as a result of running the inference 
engine 
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Application adds and/or removes activation 
conditions from the list of activation conditions 
stored by the multimodal engine 
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Modality component performs action as a result 
of its activation condition being selected 
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Modality components put results of user 
interactions on the multimodal engine's shared 
memory area 
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